{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How TensorFlow Works"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Introduction\n",
    "\n",
    "Google open-sourced TensorFlow in November 2015.  Since then, TensorFlow has become the most starred machine learning repository on Github. (https://github.com/tensorflow/tensorflow)\n",
    "\n",
    "Why TensorFlow?  TensorFlow's popularity is due to many things, but primarily because of the computational graph concept, automatic differentiation, and the adaptability of the Tensorflow python API structure.  This makes solving real problems with TensorFlow accessible to most programmers.\n",
    "\n",
    "Google’s Tensorflow engine has a unique way of solving problems. This unique way allows for solving machine learning problems very efficiently. We will cover the basic steps to understand how Tensorflow operates. This understanding is essential in understanding recipes for the rest of this book."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How TensorFlow Operates\n",
    "\n",
    "At first, computation in Tensorflow may seem needlessly complicated.  But there is a reason for it: because of how Tensorflow treats computation, developing more complicated algorithms is relatively easy.  This recipe will talk you through pseudocode of how a Tensorflow algorithm usually works.\n",
    "\n",
    "Tensorflow is now supported on all three major OS systems (Windows, Linux, and Mac). Throughout this book we will only concern ourselves with the Python library wrapper of Tensorflow. This book will use Python 3.X (https://www.python.org) and Tensorflow 0.12+ (https://www.tensorflow.org).  While Tensorflow can run on the CPU, it runs faster if it runs on the GPU, and it is supported on graphics cards with NVidia Compute Capability 3.0+. To run on a GPU, you will also need to download and install the NVidia Cuda Toolkit (https://developer.nvidia.com/cuda-downloads).  Some of the recipes will rely on a current installation of the Python packages Scipy, Numpy, and Scikit-Learn as well.\n",
    "\n",
    "Please see the `requirements.txt` in the main directory of this repository and run a command similar to\n",
    "\n",
    "`pip install -r requirements.txt` to guarentee that all the necessary libraries are available."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### General TensorFlow Algorithm Outlines\n",
    "\n",
    "Here we will introduce the general flow of Tensorflow Algorithms. Most recipes will follow this outline.\n",
    "\n",
    "1. Import or generate data.\n",
    "\n",
    "    All of our machine learning algorithms will depend on data.  In this book we will either generate data or use an outside source of data. Sometimes it is better to rely on generated data because we will want to know the expected outcome. Other times we will access public data sets for the given recipe and details on accessing these are in section 8 of this chapter.\n",
    "\n",
    "2. Transform and normalize data.\n",
    "\n",
    "    The data is usually not in the correct dimension or type that our Tensorflow algorithms expect.  We will have to transform our data before we can use it.  Most algorithms also expect normalized data and we will do this here as well.  Tensorflow has built in functions that can normalize the data for you.\n",
    "\n",
    "    `data = tf.nn.batch_norm_with_global_normalization(...)`\n",
    "\n",
    "3. Set algorithm parameters.\n",
    "\n",
    "    Our algorithms usually have a set of parameters that we hold constant throughout the procedure.  For example, this can be the number of iterations, the learning rate, or other fixed parameters of our choosing.  It is considered good form to initialize these together so the reader or user can easily find them.\n",
    "\n",
    "    `learning_rate = 0.01`\n",
    "    `iterations = 1000`\n",
    "\n",
    "4. Initialize variables and placeholders.\n",
    "\n",
    "    Tensorflow depends on us telling it what it can and cannot modify.  Tensorflow will modify the variables during optimization to minimize a loss function.  To accomplish this, we feed in data through placeholders.  We need to initialize both of these, variables and placeholders with size and type, so that Tensorflow knows what to expect.\n",
    "\n",
    "    `a_var = tf.constant(42)`\n",
    "    `x_input = tf.placeholder(tf.float32, [None, input_size])`\n",
    "    `y_input = tf.placeholder(tf.float32, [None, num_classes])`\n",
    "\n",
    "5. Define the model structure.\n",
    "\n",
    "    After we have the data, and initialized our variables and placeholders, we have to define the model.  This is done by building a computational graph.  We tell Tensorflow what operations must be done on the variables and placeholders to arrive at our model predictions.  We talk more in depth about computational graphs in chapter two, section one of this book.\n",
    "\n",
    "    `y_pred = tf.add(tf.mul(x_input, weight_matrix), b_matrix)`\n",
    "\n",
    "6. Declare the loss functions.\n",
    "\n",
    "    After defining the model, we must be able to evaluate the output.  This is where we declare the loss function.  The loss function is very important as it tells us how far off our predictions are from the actual values.  The different types of loss functions are explored in greater detail in chapter two, section five.\n",
    "\n",
    "    `loss = tf.reduce_mean(tf.square(y_actual – y_pred))`\n",
    "\n",
    "7. Initialize and train the model.\n",
    "\n",
    "    Now that we have everything in place, we create an instance or our graph and feed in the data through the placeholders and let Tensorflow change the variables to better predict our training data.  Here is one way to initialize the computational graph.\n",
    "\n",
    "    `with tf.Session(graph=graph) as session:\n",
    "\t...\n",
    "\tsession.run(...)\n",
    "\t...`\n",
    "\n",
    "    Note that we can also initiate our graph with\n",
    "\n",
    "    `session = tf.Session(graph=graph)`\n",
    "    `session.run(…)`\n",
    "\n",
    "8. (Optional) Evaluate the model.\n",
    "\n",
    "    Once we have built and trained the model, we should evaluate the model by looking at how well it does on new data through some specified criteria.\n",
    "\n",
    "9. (Optional) Predict new outcomes.\n",
    "\n",
    "    It is also important to know how to make predictions on new, unseen, data.  We can do this with all of our models, once we have them trained."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary\n",
    "\n",
    "In Tensorflow, we have to setup the data, variables, placeholders, and model before we tell the program to train and change the variables to improve the predictions.  Tensorflow accomplishes this through the computational graph.  We tell it to minimize a loss function and Tensorflow does this by modifying the variables in the model.  Tensorflow knows how to modify the variables because it keeps track of the computations in the model and automatically computes the gradients for every variable.  Because of this, we can see how easy it can be to make changes and try different data sources.\n",
    "\n",
    "Overall, algorithms are designed to be cyclic in TensorFlow.  We set up this cycle as a computational graph and (1) feed in data through the placeholders, (2) calculate the output of the computational graph, (3) compare the output to the desired output with a loss function, (4) modify the model variables according to the automatic back propagation, and finally (5) repeat the process until a stopping criteria is met."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Tensorflow_Algorithm_Diagram](https://github.com/nfmcclure/tensorflow_cookbook/raw/jupyter_notebooks/01_Introduction/images/01_outline.png)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
